1. Field
The present invention relates to the field of human interaction with mobile devices, more specifically the interaction done by the automatic recognition of barcode images captured by the camera integrated in portable devices, particularly mobile phones, smartphones, Personal Digital Assistants PDAs, portable digital camera, among others.
The present patent application relates to a method to effect the reading of barcodes by means of portable devices with digital cameras. The present invention provides a method to restore barcode images damaged by the blurring effect due to the lack of focus of the captured images with low cost cameras with fixed focus. The proposed method treats the image of barcodes in order to enable correct decoding of its content.
2. Description of the Related Art
Currently, barcode technology is part of various sectors of the commercial, industrial and financial. We find barcodes to identify the products of a supermarket, a library of books on machine of an industry, in bank bills, equipment of an organization, employee badges in a company, etc.
In general, barcodes representing unique information via the widths of bars and spaces are arranged in a parallel horizontal pattern. This type of barcode has only one dimension and is known as one-dimensional barcode or linear barcode. An example of linear barcodes includes the EAN-13, Code 128, and Code 39, among others.
More recently, barcodes have emerged that are capable of storing information in vertical and horizontal patterns, and are called 2D barcode or two-dimensional barcodes. They can store a lot of information. As an example we can mention the QR Code, DataMatrix, PDF417, and Code 49, among others.
To perform the reading of the barcode, optical reading equipment based on what is known as a scanner or barcode reader is used. The patent document US 20110068173A1 entitled SYSTEM AND METHOD FOR RECOGNIZING DEFORMED LINEAR BARCODES FROM A STREAM OF VARIED-FOCUS VIDEO FRAME, published on Mar. 24, 2011, proposes a system for recognizing barcodes. This system performs the reading of multiple frames shot by the camera in the preview mode, wherein at least one of these tables must have a barcode. This system US20110068173A1 document tries to estimate the level of blurring caused by lack of focus in video frames. This process also tries to estimate the identity of the barcode. The method taught by document US20110068173A1 locates the region containing the barcode. Another operation performed by the document US20110068173A1 is to create a geometrical model barcode with deformities found in the barcode, which, in turn, is used to restore the barcode. These deformities can be caused by the curvature of the packaging, the level of tilting the camera, and ambient light, among other factors. However, the cited patent has no details on creating this geometric model. Document US20110068173A1 also estimates the level of blurring, where the blurring level is high, the probability of success in decoding the barcode this frame will be low, and this, in turn, is dropped. This estimation is performed by calculating the differences between frames filmed and by the same, it attempts to obtain the radius of blur present in the barcode, where this threshold is less than the predetermined defocus then is effected reading the barcode. The document US20110068173A1 does not present a method that is able to restore the blur in the picture of the barcode, it just checks if the image has an ideal focus, otherwise it discards the same. With this, only the barcodes present in sharp images are decoded.
The international patent application WO2011011051A2, IMAGE-BASED BARCODE READER, published on Jan. 27, 2011, discloses a method that performs the decoding of barcodes present in images from deformable models, where from the parameterization of these attempts through the maximum similarity obtain the digits of the barcode without turning into bits the image with the barcode and using all the pixel information in grayscale. For decryption to be successfully performed, it is necessary that the location of the barcode is within a reasonably precise location where a tolerance is equal to twice the width of the thinnest bar at either end. Because the sizes of labels on which barcodes are printed do not have a pattern, this method may limit the decoding of barcodes in situations where the widths of the bars do not match the submitted location tolerance. Therefore, the solution proposed in International Patent Application WO2011011051A2 is not adaptive with respect to decoding barcodes with varied geometries. To perform the decoding of the barcode, the method proposed by said patent application WO2011011051A2 tries to obtain digits comparing pixels of the image of the barcode with predetermined templates, where they are scaled and translated to resemble those digits into analysis. The decoded digit is the one with the highest likelihood. This process may be slower than the use of filtering, since the process of obtaining the digit corresponds to the model is computationally expensive.
The patent document US20110007967A1 entitled: METHOD AND ARRANGEMENT FOR RETRIEVING INFORMATION COMPRISED IN A BARCODE, published on Jan. 13, 2011 presents a method that returns information comprised in a barcode. This method analyzes the first image to see if it has a barcode, and checks the quality of this region where the barcode is located. If the first picture has a barcode, then a second image is acquired, where the quality of this image is greater than the first image and the second region with the barcode overlaps at least a portion of the first region. Then, the decoding is effected with the barcode from the second image. The document US20110007967A1 tries to find the barcode on a first image with low quality. To locate the barcode, a search is conducted for rectangular regions in the image using a sliding window. The following extract is then associated with the barcode and the method SVM (Support Vector Machine) is performed to evaluate whether or not that window has a barcode. If this is found, then an image with high quality is obtained. This is achieved by using quality improvement techniques such as auto-focus, flash, and lighting techniques, among others. Said document US20110007967A1 presents no method for restoring a damaged image nor proposes an adaptive method for obtaining the information represented by the barcode.
The Brazilian patent document PI0603273A, entitled System and Method that uses a Software to Perform Decoding Barcodes Directly on Mobile Communication Devices Equipped with Digital Camera, published on Jan. 15, 2008, describes a method for decoding a barcode from images captured by a digital camera of mobile communication devices. This sends the result of decoding to a server via a network connection, which, in turn, will provide product information related to the number represented by the decoded barcode. If there is a network connection, one can store the result of the decoding for future reference. The decoding is performed in real time by analyzing the intercepted frames in the preview mode, and can also be performed by capturing a high resolution image from the button press. After getting the image with the barcode, it begins the process of image processing, recognition and decoding. The Brazilian document PI0603273A proposes methods for turning the image into bits, correct lighting and remove noise, but does not present details on the operation of these. This document also does not present a solution for restoration of blurred images and has no adaptive methods for converting image information represented in the barcode.
The American document US007237721B2 entitled: IMAGE PROCESSING FOR PATTERN DETECTION published on Jul. 3, 2007 describes a method that detects patterns such as barcode in an image. This method finds the points of starting and ending patterns in a region of an image, tries to estimate the PSF (Point Spread Function—Point Spread Function (PSF) that is the degree to which an optical system blurs (spreads) a point of light. PSF is the inverse Fourier transform of the transfer function of optical frequency domain) from the start and end points detected, and restores the region of the image using the PSF obtained. The method proposed in this document US007237721 B2 assumes that the PSF is Gaussian, which is not always correct and that may limit the restoration of the barcode. Furthermore, the method performs deblurring PSFs with 1 L along scan lines, but the PSF to restore an image should be 2 L, which limits the power in the information retrieval barcode. Another limitation of the proposed method is the use of a PSF with parameters determined for the picture, but the parameters of PSF should vary over the image to perform a reconstruction closer to the ideal.
The document US20060202039A1 entitled CAMERA-BASED BARCODE RECOGNITION), published Sep. 14, 2006, discloses a system for identifying a barcode using digital cameras. The document US20060202039A1 uses methods that reduces noise and increases the contrast between the bars, but it does not describe in detail how these operations are performed. It provided a method for locating the barcode in any orientation and position, but for this, it is necessary that the barcode is framed in the center of the image. The method of locating document presented by US20060202039A1 is inefficient due to the high computational cost and cannot identify the barcode in degraded images. For turning the image into bits, the document US20060202039A1 does not use adaptive methods, causing the captured image to be in optimum illumination for turning into bits efficiently. The document also does not propose a method for restoring blurred images.
The American US20050011957A1 document entitled: SYSTEM AND METHOD FOR DECODING AND ANALYZING BARCODES USING A MOBILE DEVICE, published on Jan. 20, 2005, proposes a system and method for decoding barcodes. This document provides a system and method that attempts to improve the image captured by a camera integrated into the mobile device and then tries to get the information represented by the barcode present in this image. The information obtained is sent to a server through a wireless connection, which performs a query on the content corresponding to the number decoded. This content, in turn, is transmitted to the mobile device. Document US20050011957A1 shows a block diagram describing a process used to correct the image and decoding the barcode, but does not detail the procedures shown in this block diagram. The procedure described in that document gets the pictures filmed by the camera integrated into the mobile device and from these numerous attempts performs decoding with different parameters until the barcode is successfully decoded or until all parameters are used. The document mentions US20050011957A1 methods to remove distortions which may damage the image of the barcode and, in turn, hinder the decoding thereof. These methods are used for: perspective correction, removal of shear; correcting problems associated with lighting, and adjustable focus, among others. However, no details are given about the operation of these methods, making it not possible to make an evaluation. The document does not describe US20050011957A1 step for obtaining the cut-off threshold between black and white, which is used in the conversion process for the pixels bars. This document does not mention any method to find or identify in advance the barcode before decoding, it does not detail the process of getting the bars and it does not present a method for restoring blurred images.
The article “Robust Recognition of 1-D Barcodes Using Camera Phones” Wachenfeld Steffen et al., Published in 2008 in IEEE, describes an algorithm for decoding barcodes using camera phones. The algorithm described by the article cited in this paragraph uses the techniques of image processing such as Hough transforms, wavelets to localize the barcode patterns and morphological operations. However, this article does not describe any method for restoring images barcode damaged by the effect of blurring, and only details the procedures used in localizing the barcode, digitizing the scan line and identify digits represented thereby. In turn, such procedures have been shown to be ineffective in decoding barcodes present in damaged images, because the proposed procedures are not adaptive enough, making it less efficient for decoding.
The article “A Robust Method for Blind Deconvolution of Barcode Signals and non-uniform Illumination Estimation,” L. Dumas et al., 17th IEEE International Conference on Image Processing (ICIP), September 2010, Hong Kong, available via the link http://www.math.uvsq.fr/˜dumas/Proc8.pdf, proposes a method that restores the image of the barcode by removing noise and blurring using blind deconvolution. The method proposed by the article tries to remove distortions caused by non-uniform illumination. This method is based on a genetic algorithm that combines optimization applied to discrete and continuous data strongly affected by the presence of noise and blurring effect. The article describes a method for blind deconvolution based on a genetic algorithm whose solution is a target binary image, i.e., the parameters of the PSF, illumination and adjustment are also estimated automatically so as to minimize a cost function. Note that the uncertainty in the speed of convergence and the processing time of genetic algorithms may limit its applications in portable devices.
The article “Deblurring and Restoration in Barcode Signal Processing”, Todd Wittman, published in 2004 in SIAM CONFERENCE ON IMAGING PROCESSING 2004, available through the link http://www.math.ucla.edu/Wittman˜/thesis/bars_poster. pdf, presents a technique of removing blurring based on the minimization of the total variation for restoring images of barcodes. The approach can also be classified as blind deconvolution, because estimates of the parameters of the PSF and barcode together. The minimization of a cost function is performed using classical optimization methods, such as Newton and “steepest descent”, which though more efficient than genetic algorithms, suffer from the limitations of local methods and also imply a computational cost that may limit its applicability in portable devices. Another limitation of this method is to assume that the PSF is a Gaussian function 1D, which is not always true.
The article “Digital Image Processing Barcode using Mobile Phones” James Tudisco Henrique dos Santos and Almir Olivette Artero, published in 2010 at the Sixth Workshop on Computer Vision, July 2010, held in Presidente Prudente, São Paulo, proposes the use of mobile phones with integrated digital cameras to decode barcode EAN-13. The article describes a method to locate and decode barcodes. The Sobel operator is used for edge detection to locate the barcode and calculate the direction of the edges from the gradients. This article presents no method to correct problems related to lack of focus, problems associated with lighting, and presence of noise, among others. The turning into bits described in this article may be ineffective because it has a fixed threshold and thus the results of this can be harmed under different lighting conditions.
The article “Locating and Decoding EAN-13 Barcodes from Images Captured by Digital Cameras” by Douglas Florian and Chai Hock, 2006, proposes a vision-based technique to locate and decode barcodes. The article mentioned in this paragraph presents no technique for treating distortions in the image, and this has emphasized the procedure for locating the barcode.
The above documents have methods that perform the decoding barcode and restore damaged images due to the presence of noise, problems related to the focal length, the problems associated with ambient lighting, low contrast, and perspective distortions, among others. Due to these problems, it is complex to perform decoding of barcodes from images captured by low cost cameras because, due to the inferior quality of these cameras, images captured easily present the problems cited. Most of the documents cited above can perform the decoding of barcode under certain circumstances. However, for the decoding to be successful, it is necessary that the image with the barcode is above an acceptable threshold. Some of those documents mention the need to carry out a pretreatment on the captured image for decryption to be successful. But these papers do not detail the procedures necessary to restore the image, just adding a sort of black box indicating the need to improve the quality of the captured image. One of the solutions of the prior art is to estimate the level of blurring, where decoding is only performed if this is smaller than a predetermined threshold.
In contrast, the present invention provides a solution to technical details can restore images damaged by the effect of blurring and provides a robust method for obtaining the number represented by the barcode present on the captured image.
The solutions embodied by the present invention make it possible to decode the barcode present on images taken by cameras which have low cost, and fixed focus. The present invention has a camera calibration procedure that creates a base of PSF's. These reflect precisely the incidence of the effect of blurring at a certain point, from a certain distance. The PSF's in this database will be used in the operation of deconvolution that removes the blurring effect and restore the image with the barcode. The present invention has adaptive methods for obtaining the bars so as to provide greater robustness under varying conditions. The decoding of the barcode carries out a validation procedure and blending between different results, which, in turn, enhances the probability of success in getting the numbers represented by the barcode.